On numerical optimization theory of infinite kernel learning

In Machine Learning algorithms, one of the crucial issues is the representation of the data. As the given data source become heterogeneous and the data are large-scale, multiple kernel methods help to classify “nonlinear data”. Nevertheless, the finite combinations of kernels are limited up to a finite choice. In order to overcome this discrepancy, a novel method of “infinite” kernel combinations is proposed in [12,16] with the help of infinite and semi-infinite programming regarding all elements in kernel space. Looking at all infinitesimally fine convex combinations of the kernels from the infinite kernel set, the margin is maximized subject to an infinite number of constraints with a compact index set and an additional (Riemann-Stieltjes) integral constraint due to the combinations. After a parametrization in the space of probability measures, it becomes semi-infinite. We adapt well-known numerical methods to our infinite kernel learning model and analyze the existence of solutions and convergence for the given algorithms. We implement our new algorithm called ”infinite” kernel learning (IKL) on heterogenous data sets by using exchange method and conceptual reduction method, which are well known numerical techniques from solve semi-infinite programming. The results show that our IKL approach improves the classifaction accuracy efficiently on heterogeneous data compared to classical one-kernel approaches. This study is partially done at Instiute of Applied Mathematics, Middle East Technical University, Ankara, Turkey and Faculty of Engineering and Natural Sciences, Sabancı University, Istanbul, Turkey S. Özöğür-Akyüz Department of Mathematics and Computer Science, Bahçeşehir University, Istanbul, Turkey Tel.: +90 212 381 0310 E-mail: [email protected] G.-W. Weber Institute of Applied Mathematics, Middle East Technical University, METU, Ankara, Turkey Faculty of Economics, Management Science and Law University of Siegen, Germany E-mail: [email protected]